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Abstract 

This article examines dilemmas in measuring the impact of mentoring on teacher trainees and on their 
learners in the lifelong learning sector (LLS). In this small-scale research project, five mentors and five 
mentees were asked how they might evaluate the impact that mentoring might have, not just on 
trainees, but on their learners. The research participants, who were from different London colleges which 
all run a University accredited initial teacher education course, were asked: how might they measure 
mentoring impact? To what extent might these forms of evaluation be considered valid and reliable? The 
implementation of formal mentoring for teacher trainees in the lifelong learning sector has increased 
the need for systematic evaluation of mentoring schemes by universities and colleges in initial teacher 
education. The mentors' and mentees' suggestions for evaluating the impact of mentoring comprised 
quantitative and qualitative methods and also illustrated the significant challenges to evaluating, with any 
precision, the benefits of mentoring in hard statistical terms. 
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Context 

"At its best, mentoring can be a life-altering relationship that inspires mutual growth, learning 
and development. Its effects can be remarkable, profound and enduring; mentoring relationships 
have the capacity to transform individuals, groups, organisations and communities" (Ragins and 
Kram, 2007:3). 

The mentoring of teacher trainees in the lifelong learning sector is the focus of much current debate, partly 
because of its newly pivotal role following the reforms to initial teacher training in the lifelong learning 
sector of recent years (DfES, 2004:7) and partly because the mentor-mentee relationship itself is under 
scrutiny by both education and industry, both of which set great store by the potential of mentoring. In 
attempting to meet the need for workforce development and professionalisation, the government and 
Office for Standards in Education (DfES, 2004; Ofsted, 2003) have promoted mentoring not only as an 
effective method of advice and support for novice teachers, but as their cornerstone policy for the support 
of subject pedagogy. The mentor is thus the subject specialist who supports the mentee (teacher trainee) in 
their workplace and who observes and gives feedback on their teaching practice. 

However, the literature on mentoring in initial teacher education has tended to concentrate on: 

• the reciprocal and mutual effects of the mentor-mentee relationship; 

• theories and models of mentoring; 

• mentor and mentee roles; and 

• the feelings provoked by mentoring. 
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There seems to be little focus in the literature relating to mentoring in the LLS on the impact that 
mentoring may have on mentees' learners (i.e. the impact on the trainees' own students). This small-scale 
action research project examines possible methods of evaluating the impact of mentoring and questions 
the validity and reliability of these methods. 

The research was undertaken within a south of England University consortium which comprises seven 
colleges in London running teacher education courses for the post-compulsory sector for over 300 
participants. 

Research objective 

The objective was to collate case studies of mentors' and mentees' views on what might constitute valid 
and reliable evidence of the impact of mentoring on trainees' learners, especially in work-based learning 
and adult and community learning. This was a starting point from which we could raise questions of a 
wider nature. 

There are about 80 mentors and over 300 teacher trainees across the University's initial teacher training 
consortium. In line with the national picture of mentoring in post-compulsory teacher education, the 
provision, training and quality assurance of mentoring have been identified as priorities by the University. 

As consortium programme leader, the researcher wanted to build up an evidence base of the possible 
impact of mentoring and to extract insights that could inform improvements to the consortium's mentoring 
scheme and mentor training (Klasen and Clutterbuck, 2007:294). 

It is important to explore whether robust evidence can be gathered about the impact of mentoring on 
trainees' learners, rather than just on mentees, for several reasons: 

• to evaluate whether mentoring can improve the learning experience of the key stakeholders in 
the sector: the trainees' learners. All the mentors interviewed said that the case for mentoring 
would be stronger if we could show that learners (and not just mentees) benefit directly from 
mentoring in initial teacher education; 

• to provide insightful information about mentoring and to raise its profile as a developmental 
process since it is important that managers in the sector realise the impact and significance of 
mentoring in order to develop a 'mentoring architecture' (Cunningham, 2007:83) in partner 
colleges; 

• to inform institutions which need to fund and build mentoring capacity to comply with Ofsted 
requirements for the provision of subject-specific mentors, and to support participants on teacher 
education courses; and 

• to remind mentors and mentees how much mentoring can help them achieve (Klasen and 
Clutterbuck, 2007:295). 

Theoretical framework 

There is evidence that mentoring can have a positive impact on mentees' self-confidence, competence and 
effectiveness (Noe, 1988:459). Mentors and mentees themselves have their own ideas about the impact 
of mentoring and a start has been made to capture these ideas in the small number of case studies. The 
researcher's action research approach to the project related closely to the point of view of 'appreciative 
inquiry' (Cooperrider and Srivastva, 1987:131) because: 

• the mentors and mentees are committed to mentoring; and 

• they appreciate its potential for psychosocial and career support and its developmental nature. 
The researcher's questions aimed to identify good practice and to encourage positive thinking about 
mentoring, rather than taking a problem-solving approach - which may have led to a fixation on the 
problematic aspects of mentoring in the sector. 
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Another theoretical perspective which may throw light on the research is 'personal construct theory' 
(Cohen, Manion and Morrison, 2003:337) which school psychologist George Kelly (1955) proposed 
when dealing with problem children referred to him by teachers. Kelly tried to understand the teachers' 
complaints about these children by examining the way that teachers had construed their complaints. This 
led him to the view that there is no objective, absolute truth and that events are only meaningful in relation 
to the ways they are construed by individuals. 

The mentors' and mentees' positive views of mentoring, and their teaching and learning experiences, 
helped them to construct meanings around their experiences of mentoring. Rather than using a ready¬ 
made template of 'how to measure the impact of mentoring on mentees' learners', they were asked to 
provide their own constructs. These tended to equate the positive aspects of mentoring on mentees with a 
concomitant effect on their learners. 

A qualitative or quantitative approach? 

Ideally, in a quantitative approach, it would be useful to gather data on mentees' learners' progress before 
and after mentoring. But there are too many variables to make this a reliable methodology. There are 
problems with the quantitative data as it tends to be large-scale (e.g. colleges collect data on student 
cohorts and it is hard to disaggregate the effect of one lecturer on a student group). For the quantitative 
data to be reliable and valid, one would need to get down to the fine detail of individual mentees' 
retention, success and achievement results which are only accessible to line managers once results come in. 
These issues form part of the background to the research. 

The only immediately accessible quantitative data is attendance and that can only be accessed by the 
mentee themselves at a certain point towards the end of the year and before the data disappears into the 
college's management information systems, never to be accessed again by the mentee. Even if a mentor 
could access attendance figures for their mentees' classes, what would this tell them? There are a number 
of reasons why students don't attend sessions, such as family, domestic, legal or financial problems; 
rooming; and timetabling. Some lecturers are meticulous about keeping registers but some mark their 
students as in 'on time' even when they are over 10 minutes late. These figures could be as accurate as a 
lecturer typing in their own statistics themselves and at best, they might give anecdotal evidence or one-off 
snapshots of a lecturer's practice. 

However, colleges would argue that the data produced by completing electronic registers are valid and 
reliable as they are used as legal documents for funding purposes and also for student roll calls in the 
event of an emergency evacuation of a building. Nonetheless, each set of statistics needs to be looked at 
individually and in depth. Mentor and mentees' narratives are necessary to tell the story of the impact and 
benefits of mentoring. 

It is possible to glean some valid and reliable evidence by asking mentors and mentees questions about 
the impact of mentoring on mentees, but this becomes more problematic where intangible effects may be 
concerned. For example, do mentees gain confidence and capability and do they develop their teaching 
skills more rapidly because of their exposure to more experienced staff who are able to support them 
effectively and quickly? If they do, how do you measure this 'confidence and capability' and is it possible to 
evaluate what effect 'confidence and capability' have on their learners? If mentoring is one of the reasons 
for this growth in confidence, how can it be separated from the effects of initial teacher education as well 
as teaching practice on the mentee? 
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Research methodology 

The researcher felt that an action research approach was appropriate as part of her role in the consortium 
was the introduction and quality enhancement of mentoring. The intention was to use the outcomes of 
the research to help implement changes to policy and practice within the consortium, in keeping with 
the 'action research' principle. Ethical clearance was gained within the University of Westminster's ethical 
clearance process. 

Semi-structured interviews were conducted with five mentors and five mentees chosen from different 
colleges and across different subjects in order to get a cross-section of views and comments. Each college 
course leader had been asked to nominate an experienced mentor with a current mentee and as four 
of the mentors had also completed a pilot module in mentoring, prepared and led by the researcher, 
the course leaders suggested that these would make suitable interviewees as well. The fifth mentor was 
introduced by one of the course leaders as she was known as an experienced and approachable mentor. 
The researcher contacted the mentors and their mentees to explain the purpose of the research and to 
ask their permission to record each interview. The names of mentors, mentees and their institutions are 
anonymised for reasons of confidentiality and in order to create a safe place where participants could 
explore issues. 

Mentors and mentees were asked how to evaluate the intangible results of mentoring and what sorts 
of evidence could be used to judge the impact of mentoring on mentees' learners (see Appendix A and 
B). Where the interviewees said that they would use quantitative data, the researcher asked them to 
gather their own data, for example, from their own college student records system. The researcher then 
interviewed the mentors again in order to explore meanings behind the data; the challenges of gathering 
such data; and what, if any, conclusions we could draw from them. These second interviews were either 
face-to-face, by email or by telephone and were designed to follow up ideas, to probe responses and to 
investigate motives and feelings (Bell, 2005:157). 

Although the researcher took an action research approach to the project, this related more closely to the 
point of view of 'appreciative inquiry' (Cooperrider and Srivastva, 1987:131) for several reasons. 

Firstly, all the mentors and mentees without exception were committed to mentoring and were highly 
motivated to show that mentoring 'works'. 

Secondly, the 'action' did not result in an incontestable conclusion proving the impact of mentoring 
on mentees' learners. The project started from an appreciation of mentoring and progressed to raising 
fundamental questions about its impact, rather than taking the stance of problem-solving action research, 
since collating narratives does not fit easily with a measurement paradigm. The trustworthiness of the 
conclusions depends on those views and narratives, although these may contain bias, for instance: 

• because the researcher is the consortium programme leader and the mentors and mentees may have 
given her the answers they thought she wanted in order to please her; 

• they may have wished to demonstrate that their college is delivering 'good' mentoring; or 

• they may have wished to validate the mentoring process to boost the status of mentors in general. 
However, interviewing both mentor and mentee gave a certain balance and the research was informed 
by the literature on mentoring in initial teacher education. The selection of the mentors and mentees also 
represents a sample bias because the researcher asked course leaders to nominate experienced mentors 
and she already knew four of the mentors. Her good working relationships with these mentors helped 
provide a friendly, safe and open atmosphere which facilitated honest reflection and self-evaluation 

Outcomes and key learning points 

All names have been anonymised throughout. The mentor and mentee pairs are as follows: 


Mentor 

Mentee 

Doug 

Bob 

Suzie 

Nicola 

Joan 

Jagdish 

Anna 

Marie-Claire 
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Doug and Bob - construction 

Doug is a typical mentor with several roles apart from teaching: subject learning coach, school link co¬ 
ordinator, short course co-ordinator, textbook writer and part-time, unpaid master plasterer for English 
Heritage. His priority for his mentee Bob was to help him make the transition from on-site plasterer to 
college lecturer, avoiding what might be considered to be a common characteristic of new vocational 
lecturers: a practical approach that eschews all theory and an impatience for those who don't 'get it' the 
first time. 

Building sites are not a friendly place to be at times...I had found myself talking to [college] students like 
they were one of the boys on the building site and that's a big no-no. You can't give [students] a clip round 
the ear if they get something wrong (Bob q8). 

(The number following Bob's quote refers to the transcript of the interview with Bob and this note appears 
on line 8 of that transcript. Later quotes use similar notation). 

Doug felt that an important part of his role was to help the mentee to move on from being a somewhat 
impatient apprentice teacher with classroom favourites and a dislike of theory, towards forging a new 
identity as a professional lecturer in his new community of practice (Lave and Wenger, 1991). 

You've got to get teachers to think like teachers (Doug q 166). 

This caused disagreements between them in the beginning when Bob showed his students how to speed 
up their work, without teaching them any theory or correct procedures first because 'that's how it's done 
on site'. Some students felt that Bob didn't like them and didn't have the patience to teach them if they 
grasped things more slowly than others, and the result was that these students started to fall behind. 

Doug team-taught with Bob, who showed surprise at the amount of preparation needed for one class. 
Using mentoring skills, such as questioning and re-framing, Doug demonstrated the importance of 
differentiation, inclusion, ground rules, tracking and embedding numeracy and literacy into sessions. 

When asked how mentors could measure the impact of mentoring on learners, Doug said that success 
and achievement rates for the plastering group that he teaches with Bob had risen year on year from 71 % 
to 91 %, which he thinks may partially be due to the team teaching (which formed part of the mentoring 
activities) and partially to Bob's input in the practical sessions (which did not). However, the retention, 
achievement and success data for all plastering groups have improved overall (see Table 1). 



Plastering classes 



2005/06 

2006/07 

2007/08 

Retention 

73% 

80% 

88% 

Achievement 

87% 

96% 

93% 

Success 

64% 

76% 

81% 


Table 1. Effects of mentoring in plastering group. 


Quantitative measurements would seem to be the most unambiguous way to judge the impact of 
mentoring, but these can be problematic. It is not always straightforward for mentors or mentees to access 
statistics that relate to one mentee only, as statistics tend to relate to student cohorts rather than individual 
teachers. Even where these data can be accessed, they need to be compared with like-for-like results and 
the data analysis needs to allow for variables. For example, rooming or equipment can improve or get 
worse year by year; other lecturers in the team can have an effect on learners' morale, their progress, their 
motivation and their results. 

Other questions need to be raised about the evaluation of mentoring impact within institutions: for 
example: 

• Who should collate this data - the mentor, the mentee or the line manager? 

• What implications does this have? 

• What should be done with the data? 
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Doug said that the increased attendance and progress of some students with specific learning difficulties 
might be partially down to his mentee Bob's increasing individual attention and good working relationships 
with the learners. For example, Bob managed to get an apprenticeship for a demotivated ESOL student 
with dyslexia and behavioural problems and the student went on to win a regional plastering competition. 
Although Bob is pleased with the progress made by his learners, he constructs mentoring impact on 
learners with reference to his own personal development. He highlights the emotional tension in the 
transition from his role on the periphery of teaching to fitting in with college norms while remaining true to 
himself (Cain, 2009:56), rather than providing a logical analysis of the impact on learners: 

From that very first day when I stood in front of the students feels like a million miles - and I have 
a million miles to go. Confidence is the main booster (Bob q77). 

Suzie and Nicola - hairdressing 

When Suzie first started mentoring, she didn't consider the benefits for learners to be as important as 
the benefits for the mentee. With experience and mentor training, she sees the mentee as the channel 
for learners' success. She looks for evidence of successful mentoring in comparisons before and after 
mentoring between learners' individual learning plans (ILPs), tracking learners' progress and mentees' 
observed teaching practice. She also identifies important but intangible benefits of mentoring for her 
mentee. 

The relationship within the group with her and her learners has really developed. It was 
shambolic last year. She was new and they ruled the roost and they have their ways and didn't 
want to do this or that. She went along with what they said. This year it's completely different. 
She has control and the balance is much better (Suzie q119). 

Suzie's mentee Nicola believes that the impact of mentoring on her learners has been dramatic and she 
collated the results in Table 2 to demonstrate the constructive collaboration between her and her mentor. 
She believes that these results are self-evident proof that mentoring works - but this cannot be separated 
from the effect of another year's teaching experience and from two years' teacher training. Nonetheless, 
Nicola's personal construction of mentoring is that it played a large part in her progress, which fits in with 
her view of the teacher as the pivotal influence in students' learning. She had expected mentoring to be 
a combination of personal life coaching and therapy centred on the lecturer but she now feels that the 
mentoring also helped her improve her learners' achievements. 

It's had a knock on effect for my students. I'm much more confident. Before I met [Suzie] my 
confidence [as a teacher] was 0.5 out of ten. But now it's 9/10. What you want is someone to 
hear you; to hear the unspoken word (Nicola q165). 


Students 2007/08 

(before mentoring) 

3 competent (43%) 

4 not yet competent (57%) 


Students 2008/09 

(after mentoring) 

7 competent (78%) 

2 not yet competent (22%) 


Table 2. Effects of mentoring for hairdressing group. 
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Joan and Jagdish - tutorials 

Mentoring can be full of contradictions (Cain, 2009:53) as Joan found when she was asked to mentor 
Jagdish. Jagdish is a student learning adviser who does not have a subject specialism as she runs large 
numbers of tutorials with paper-based resources. As an experienced lecturer, tutor and E-Champion for 
teacher training, Joan focussed on the practice of tutorials that became Jagdish's 'subject specialism'. 

Joan and Jagdish agreed to use technology to overcome a number of barriers: 

• large numbers of students at Levels 1, 2 and 3; 

• lack of time allocated for individual tutorials; 

• heavy workload including university reference reports, learning reviews and progress reports for 
each student, individual help with assignments; and 

• liaison with all subject lecturers and parents. 

Joan helped Jagdish to become more efficient through setting up individual tutorials online with links 
to websites, more inspiring resources, interactive quizzes and a discussion forum so that students could 
complete their ILP targets and tutorial work online in their own time. This released some time for personal 
tutorials. Jagdish learnt to make more advanced use of the virtual learning environment (VLE) management 
information systems, new technology and electronic resources. 

Given the time constraints on Jagdish, Joan suggested she focus on Level 1 students who are generally 
most in need of tutorial support. Jagdish was able to track the progress of these Level 1 learners more 
closely and this made it easier to ensure that they all completed their first semester modules on time. Joan 
could track the increased student use of the VLE by extrapolating statistics for one of Jagdish's classes 
every week and breaking them down into types of resources accessed, dates, the number of uses and the 
number of students (see Figure 1). This table shows the use of different aspects of the college's Virtual 
Learning Environment (VLE); for example, the students' use of Hot Potatoes software, how many students 
viewed resources, how many students looked at course information, used a discussion forum and uploaded 
their assignments on to the VLE. But, as she points out, these do not state what type of resources were 
looked at or whether they were used at all or whether students looked at only one resource or a whole 
range of resources. 


Figure 1. Use of Virtual Learning Environment 

VLE use 


24.3.09 31.3.09 28.4 09 12.5.09 19.5.09 2.6.09 

Date 


Joan sees the greatest achievements of mentoring as improved time management, a more manageable 
workload once tutorials became accessible online and the growth in Jagdish's self-confidence. 


■ hotpot 

■ resource uew 
course view 

j forum 

■ assignment 
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Thematic discussion 

Most of the mentors interviewed said that in their opinion the major impact of mentoring seems to lie in: 

• better learner behaviour; 

• improved classroom management; 

• increased student participation and motivation; and 

• focus on students' learning rather than the lecturer's teaching. 

Although quantitative data can be relatively straightforward to collate and may seem unambiguous 
the qualitative reflections of mentors and mentees can produce a more nuanced interpretation of the 
relationships at the heart of mentoring: those of mentor and mentee; mentee and their learners. As Miles 
and Huberman state (1984), dangers lie in the "over-preoccupation with method rather than substance". 
Becoming a teacher requires more than absorption of teaching standards and competences. It requires 
a social process which Lave and Wenger (1991) describe as "legitimate peripheral participation". The 
mentees do not yet feel part of their 'college teacher' group, yet they are not disinterested observers. 

Their position is peripheral because to some extent they remain on the outside looking in, until the mentor 
introduces them into what it means to become a teacher in the LLS. 


Anna and Marie-Claire - travel and tourism 

Mentoring impact might also be seen in significant differences in the results of formative assessment 
according to Anna, a mentor who specialises in travel and tourism. She followed up an initial observation 
of teaching practice by jointly planning the next observed teaching session with her mentee with the 
aims of improving the mentee's differentiation and questioning; breaking down the amount of material 
she was going to cover into more palatable stages; and arranging more carefully the composition of 
classroom groups. The mentee felt that this would have a snowball effect on her teaching - although she 
said that she couldn't put that amount of time into planning every lesson. Anna then asked her to count 
up the number of students with referred work (pieces of coursework which needed to be re-written and 
re-submitted for marking) after their jointly planned teaching session and to compare that figure with the 
average number of pieces of referred work she usually had to re-mark. Out of 40 students in a normal 
lesson, Marie-Claire normally referred between 10 to 15 pieces of work (approximately 25% to 30%). Out 
of 40 students in the jointly planned session, only 4 students (10%) were referred. 

Preparation time for a normal lesson would be approximately two hours whereas I would say 
the observed lesson took us about four hours to plan, but as we teach this unit every year, the 
material will always be used. The time saved in marking the work would be around two hours 
(Anna q64). 

Anna therefore felt that as her mentee would be repeating this class year after year, she might be saving 
marking time and also setting herself a higher standard when planning future sessions. 

Conclusion 

Each of the short case studies relates to a different discipline and each mentor chooses to focus on a 
different type of data: 

• In construction the mentor compares retention, achievement and success data (mentor: Doug; 
mentee Bob); 

• In hairdressing the mentor compares the proportion of competent/not yet competent learners 
between her first and second years of teaching (mentor: Suzie; mentee Nicola); 

• In tutorials the mentor monitors the use of a VLE by her mentee's learners (mentor: Joan; mentee 
Jagdish); and 

• In travel and tourism the mentor calculates the percentage of assignments referred before and 
after joint lesson planning (mentor: Anna; mentee Marie-Claire). 
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Perhaps our task here is not to seek common agreement or to provide 'conclusive evidence' (Davies, 2000) 
but to raise questions and provide competing accounts and evidence. 

At the heart of mentoring is the psychosocial and career support that mentors provide. Yet it remains 
difficult to measure the effects of core mentor competences such as self-awareness, beliefs and attitudes, 
questioning, listening and self-management not just on mentees, but on mentees' learners who are at one 
remove from the mentoring itself (European Mentoring and Coaching Council, 2007). Unlike coaching, 
mentoring may have indirect, subtle and long-term effects rather than directly measurable short-term ones. 
Indeed, why should we expect the 'impact' of mentoring to be discernible immediately? 

Even industry has failed to devise a generic evaluation method for mentoring impact due to the divergence 
in different organisations' goals, the fact that evaluation focuses on measuring human beings and the 
essentially confidential nature of the mentoring relationship (Klasen and Clutterbuck, 2007:297). The 
mentors and mentees who were interviewed, however, feel that there are ways of evaluating impact on 
mentees' learners despite the challenges that this may bring. 
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